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NPS  Testbed  for  Team  Collaboration 

Model  Validation 


Objective 

•  Better  understand  cognitive 
processes  empioyed  when 
teams  coiiaborate  to  soive 
probiems 

Approach 

•  Anaiyze  team  communications 
data  using  cognitive  process 
definitions 

•  Vaiidate  and  refine  the  modei 
of  team  coiiaboration 
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Data  Anaiyzed 

•  Three  Maritime  Interdiction  Operations  (MIO)  experiments 

•  Four  Air  Warfare  scenarios 

•  Firefighters  9-11 
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Team  Collaboration  Model  Validation 


GOAL:  Understand  and  improve  effectiveness  of  team  decisionmaking  in 

complex,  data-rich  situations  by  validating  model  of  team  collaboration 
Model  of  Team  Collaboration  Defines: 

-  Meta-cognitive  processes  that  guide  team  collaboration 

-  Information  processing  components  the  team  performs  to  achieve 
each  collaborative  stage 

-  Communication  mechanisms  used  by  the  team  to  build  the 
necessary  knowledge 

-  Emphasizes  cognitive  aspects  of  collaboration  process  —  includes 
major  cognitive  processes  that  underlie  this  type  of  communication: 

(1)  Individual  knowledge  building 

(2)  Knowledge  interoperability 

(3)  Team  shared  understanding  and 

(4)  Team  consensus  (Warner,  Letsky,  &  Cowen,  2004) 

-  Validate  that  these  processes  exist  and  how  they  contribute  to 
team  performance  through  verbal  protocol  analysis  coding  of 
team  communications. 
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Types  of  Problem  Solving  Situations 


WTftASSTWiTLA  PERSQ£^rT^^^ 


l-Structured  Decisionmaking  Tasks 

•  Time  Pressure 

•  Dynamic  information 

•  High  Information  Uncertainty 

•  High  Cognitive  Workioad 


Human  System  interface 
Compiexity 


LLNL 


DISTRICT  11  (USCG) 


COALITION 

PARTNERS 


INGAPORE 


AUSTRIA 


SWEEDEN 


TNT  —  MIO  Scenario 
Multi-Level  Committee  (of  Teams) 
Decision  Support  (DS)  Structure 

Multiple  DMs  with  complete  participant  interaction  at  multiple 
levels  in  the  chain  of  command. 


☆  Represents  Decision 
Making  Node 

©Contributor/Team 
Member  of  Node 


Team  Types 


rtiAdm  Cehrowski  Network  (Entric  warfare 


VTpwSmSi;*  TEH  SCiENTJ^*,  P 


Asynchronous 
Distributed 
Culturally  Diverse 
Heterogeneous  Knowledge 
Unique  Roles 
Command  Structure 
Rotating  Team  Members 


Operational  Tasks 

*  Team  Data  Processing 

*  Developing  Shared  Situational  Awareness 

*  Team  Decisionmaking  and  Course  of  Action  Selection 
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Problem  Area 


NAV^AIR 

Office  of  Naval  Research 
Collaboration  and  Knowledge  Management  (CKM)  Program 

MODEL  OF  TEAM  COLLABORATION 

Focus  on  Macro-Cognition  (September,  2005) 


srmm 


Systems  Canlsr 
SanDisgo 


0  Characteristics 


Collaborative  Situation 


Parameters: 


•  time  pressure 

•  information/knowledge 
uncertainty 

•  dynamic  information 

•  iarge  amount  of  knowiedge 
(cognitive  overioad) 

•  human-agent  interface 
compiexity 

Team  Tvoes 


•  asynchronous 

•  distributed 

•  cuituraiiy  diverse 

•  heterogeneous  knowiedge 

•  unique  roies 

•  command  structure 
(hierarchicai  vs.  fiat) 

•  rotating  team  members 
Ooerational  Tasks 


•  team  decision  making,  COA 
seiection 

•  deveiop  shared  understandir 

•  inteiiigence  anaiysis 
(team  data  processing) 


Collaboration  Stages  &  Cognitive  Processes 


Meta-Cognitive: 

•  individual  conversion  of 
data  to  knowledge 


Macro-Cognitive: 

•  individual  mental  model 
construction 

•  knowledge  interoperability 
development 

•  individual  task,  team  and  domairi 
knowledge  development 

•  individual  knowledge  object 
development 

•  individual  visualization  and 
representation  of  meaning 


» team  integration  of  individual 


*  team  agreement  on  a  common 


knowledge  for  common  understanding  solution 


•  knowledge  interoperability 
development 

•  iterative  information  collection 

and  analysis 

•  team  shared  understanding 
development 

•  develop,  rationalize,  &  visualize 
solution  alternatives 

•  convergence  of  individual  mental 
models  to  team  mental  model 

•  individual  task,  team  and  domain 
knowledge  development 


*  solution  adjustment  to 
fit  goals  and  exit  criteria 


•  team  negotiation  of  solution 
alternatives 

•  team  pattern  recognition 

•  team  shared  understanding 
development 

•  convergence  of  individual 
mental  models 

•  critical  thinking 

•  sharing  hidden  knowledge 

•  individual  task  knowledge 
development 

•  team  task  knowledge 
development 


*  compare  problem  solution 
against  goals 

*  team  shared  understanding 
development 

*  convergence  of  individual 
mental  models  of  solutioi 

*  analyze,  revise  output 
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Mechanisms  for  achieving  Meta  and  MacroCogiritive  Processes  (applies  to  all  stages) 

•  Verbal  communications:  representing  and  discussing  individual  information,  discussing  team  generated  information, 
questioning,  agreeing  /  disagreeing,  negotiating  perspectiyes,  discussing  possible  solutions,  providing  rationale. 

•  Non-Verbal  communications:  facial  expressions,  voice  clues  (vocal  paralanguage),  hand  gestures,  body  movements(kin^i 
touch  ^aptics),  personal  space,  drawing,  text  messages,  augmented  video,  affordances(cognition  in  objects). 


:s) 


Method 


•  Verbatim  transcripts  analyzed  from  two  series  of  exp’ts  and  one  real- 
world  event  where  teams  collaborated  to  solve  a  complex  problem 

-  Maritime  Interdiction  Operations  (MIO) 

-  Air  warfare  decisionmaking 

-  Firefighters  from  9-11 

•  In  all  three  problem-solving  tasks,  assessment  is  difficult  because 
available  information  is  often  incomplete  or  ambiguous. 

-  Transcripts  include  communications  between  all  team  members 
and  decisionmakers  at  distributed  sites. 

•  Analyze  and  code  team  communications  data  using  the  cognitive 
process  definitions  developed  by  Warner,  Letsky,  &  Cowan,  2004. 

-  Focus  of  collaboration  model  is  on  knowledge  building  among 
team  members  and  developing  team  consensus  for  selection  of 
a  course  of  action 

-  Builds  on  previous  work  to  validate  model  (Warner,  et  al,  2004) 

-  Similar  methodology  applied  to  three  different  DMg  tasks 
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Experiment  I:  Maritime  Interdiction  Operations 


•  Tech’l/oper’l  challenges  of  developing  global  Maritime  Domain  Security 

-  Wireless  network  for  data  sharing  during  MIO  to  facilitate  reachback  for 
radiation  source  analysis  and  biometric  data  analysis 

-  Networking  solutions  for  MIO  where  subject  matter  experts  at 
geographically  distributed  command  centers  collaborate  with 
boarding  party  in  near  real  time  to  facilitate  SA  /  COA  selection 

•  Evaluate  networks,  adv’d  sensors,  and  collaborative  tech’y  for  rapid  MIO 

-  Rapidly  set  up  ship-to-ship  communications  that  permit  them  to 
search  for  radiation/  explosive  sources  while  maintaining  contact 
with  mother  ship,  C2  organizations,  and  collaborating  with  remotely 
located  sensor  experts 

•  Boarding  team  boards  suspect  vessel,  establishes  collaborative  networl< 
and  begins  inspections  and  data  collection  process 

-  Boarding  officer  boards  vessel  with  his  laptop  so  he  can  collaborate 
with  all  other  members  of  the  team 

-  Co-located  on  the  ship,  physically  spread  out  (searching  for 
contraband  material  and  obtaining  fingerprints  of  crew  members) 

-  Virtual  members  of  the  boarding  team  -  experts  at  reachback  centers 

-  Commercial  uses  for  certain  radioactive  sources,  positive 

identification  of  the  source  in  a  short  time  is  imperative  g 

-  Pressure  to  conduct  the  MIO  quickly  so  as  to  not  detain  the  ship 


MIO  Team  Members 


*  Members  of  the  boarding  team 

-  Boarding  Officer,  a  Coast  Guard  officer 

-  Representative  from  Lawrence  Livermore  National 
Labs  (LLNL)  with  portable  radiation  detection 
devices  and  “reach-back”  capability  to  LLNL 

-  Representative  from  the  Defense  Threat  Reduction 
Agency  (DTRA),  who  uses  biometrics 
measurements  of  fingerprints  and  video  imagery  to 
be  checked  against  databases  at  the  remote  facility 

-  Representative  from  Special  Operations  Command 
(SOCOM),  who  provides  guidance  on  handling 
hazardous  material. 
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Maritime  interdiction  Operations  Scenario 


•  US  Coast  Guard  ordered  cutter  to  stop,  board,  and  search 
commercial  vessel  of  foreign  origin  suspected  of  transporting 
uranium  enriching  equipment 

•  Boarding  party  brings  radiation  detection/  biometric  gear,  drawings 
of  dangerous  equipment  and  people,  and  video  recording  capability 

•  Data  collected  on  suspicious  material,  equipment,  and  people  and 
sent  to  specific  experts  at  distributed  reachback  centers 

•  Groove  collaborative  workspace  brought  expert  services  into  the 
boarding  party  team’s  tool  set 

-  Facilitated  voice  and  text  communications  between  all  members 
of  the  virtual  boarding  party  and  physical  boarding  party 

•  Requests,  transmitted  by  text  message  -  taken  for  action,  and 
radiation  source  spectrum  captures  were  made  of  suspect 
containers  that  were  detected  to  have  a  radiation  signature  presence 

•  Analysis  led  BO  to  recommend  vessel  be  quarantined  for  further 
inspection 

•  Biometric  team  took  digital  prints  of  the  crew  to  be  compared  to 
known  criminal  prints  and  latent  prints  from  terrorist  and  crime 
scenes 
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Air  Warfare  Decisionmaking 


•  Air  warfare  DMg  -  conducted  in  combat  information  center  of  Navy  ship 

•  Identification  of  large  number  of  air  tracks  under  high  time  pressure 

-  Multiple  hypotheses  regarding  threat  level  posed  to  the  battlegroup 
due  to  high  level  of  ambiguity  associated  with  the  data 

-  Nature  of  the  data,  complex  judgments  required,  and  socio- 
technical  environment  characterized  by  high  workload,  and  high 
stakes,  create  challenging  problem  for  the  air  warfare  team 

•  Incoming  info  arrives  via  various  sensor  systems  (radar,  electronic 
sup’t  measures  system,  identification  friend  or  foe,  etc.),  various 
reports,  e.g.,  intell,  other  platforms  in  area  pass  messages  regarding 
situation 

•  Reports  passed  to  rest  of  team  over  any  of  several  comm’s  systems 

-  Heard  by  all  team  members,  reports  typically  addressed  to 
specific  team  member/s,  sometimes  addressed  to  “all” 

-  Communications  passed  as  soon  as  information  is  received; 
updated  reports  are  passed  as  soon  as  new  information  is  obtained 

•  Reports  on  specific  tracks  interleaved  with  reports  on  other  tracks 

-  In  a  series  of  speech  turns,  five  separate  contacts  may  be 
discussed  at  various  levels  -  initial  reports,  updated  reports, 
sharing  information  on  response/  lack  of  response,  by  contact  tp 
action  taken  by  the  ship,  etc. 


Air  Warfare  Team  Members 


•  Six  collocated  team  members  consisted  of 

-  Commanding  officer  (CO) 

-  Tactical  action  officer  (TAO) 

-  Air  warfare  coordinator  (AAWC) 

-  Electronic  warfare  supervisor  (EWS) 

-  Identification  supervisor  (IDS) 

-  Tactical  information  coordinator  (TIC) 

•  Combat  information  center  team  members  also  communicate 
with  several  non-collocated  information  sources 

-  Battle  group  commander 

-  Saudi  air  tower 

-  Assets  passing  intelligence  reports 

-  Other  ships  and  friendly  aircraft  in  the  vicinity  of  the  battle 
group 

•  Gather  additional  information,  keep  them  apprised  of  the 
unfolding  scenario  as  they  collaborated  to  identify  air  tracks. 
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Air  Warfare  Decisionmaking 


•  Identification/  responding  to  numerous  air  contacts:  CiC  personnel 
work  as  a  team  to  identify/  determine  If  A/C  poses  a  threat 

-  High  ambiguity  often  makes  threat  assessment  a  very  difficult  task 

-  Many  pieces  of  data  fit  multiple  hypotheses 

-  Global  response  choices  (engage,  monitor,  do  nothing)  largely 
determined  by  ship’s  orders  and  the  current  geopoiitical  situation 

-  Specific  actions  (e.g.,  change  course,  issue  verbal  warnings, 
iiluminate  with  radar,  chalienge  with  other  sensors,  etc.)  depend  on 
local  conditions,  relative  positions  of  the  inbound  contact  and  ship 

•  Determining  which  actions  is  likely  to  be  effective  depends  on 
maintaining  an  accurate  assessment  which  requires  continually 
updating  based  on  iterative  situation  assessments 

*  Critical  contacts  ident’d  based  on  ambiguous  info,  under  time  pressure 

*  High  mental  workload  -  constant  stream  of  info,  must  be  continuously 
evaluated,  e.g.,  when  info  often  pertains  to  several  different  contacts 

*  Teams  assess,  compare,  and  resolve  conflicting  info,  make  difficult 
judgments  and  remember  the  status  of  several  evolving  situations 

•  Tasks  interleaved  with  other  tasks,  such  a  making  reports  to  higher 
authority  and  requesting  assets 

•  Situation  assessment  &  action  selection 


14 


Coding  Process 


•  Cognitive  process  coding  definitions  used  to  code  speech  turns 

•  Attempted  to  develop  criteria  for  coding  schema 

•  Codification  of  the  coding  process  is  part  of  the  overall  validation  of  the 
model,  e.g.,  goal  is  to  have  high  inter-rater  reliability  between  coders 

•  Important  to  pay  attention  to  which  track  a  team  member  was  talking  about 
when  coding  the  speech  turns 

•  First  time  discuss  a  track  -  coded  as  a  2  (individual  mental  model  (I MM) 
construction  -  where  an  individual  team  member,  using  available  info, 
develops  his/  her  mental  picture  of  the  problem  situation) 

•  After  three  speech  turns  discussing  the  same  track  (typically  involving  at  least 
four,  of  the  six  or  more  team  members)  it  was  coded  as  a  4  (team  knowledge 
development  (TKM)  -  where  all  team  members  participate  in  clarifying 
information  to  build  team  knowledge 

•  Once  five-six  team  members  had  discussed  a  track,  and  at  least  4  of  the  6  team 
members  had  been  involved  in  discussing  this  particular  track,  it  was  coded  as 
a  10  -  team  shared  understanding  development  -  which  includes  discussion 
among  all  team  members  on  a  particular  topic  or  data  item 

•  Exceptions  to  the  coding  criteria  include:  “All  stations,  [track  #  7010  is  a  comm- 
air.]”  -  he  is  telling  all  team  members  this  evaluation  of  the  track. 

-  Because  addressed  to  all  TMs  &  reported  a  higher  level/  more  final  assess’t 
of  the  track,  i.e.,  a  comm-air,  was  coded  as  a  10.  As  more  TMs  discuss 
contact  (i.e.,  more  reports  and/or  updates  have  been  shared  among  TMs), 
cognitive  process  coding  category  reflects  a  higher  level  of  team 
understanding  of  the  situation 


New  Coding  Categories 


•  Issuing  an  order  regarding  a  course  of  action  —  person  with  higher  rank 

•  Teiis  them  to  take  some  specific  action  against  a  potentiai  threat  track. 

-  Issuing  verbal  warnings,  illuminating  or  locking-on  with  radar, 
developing  a  firing  solution,  covering  with  missiles,  etc. 

-  Includes  responding/  reporting  have  taken  the  action/acknowledging 

•  Reguest  a  team  member  take  some  action  —  tell  team  member  to  do  something 

•  Not  a  direct  action  against  a  threat  track. 

-  “Can  you  try  and  change  7006  and  7005  to  assumed  hostile.” 

•  Prodding  a  team  member  to  jog  their  awareness 

•  To  make  sure  they  are  following  the  discussion 

•  Push  or  suggest  to  one  or  more  team  members  to  go  out  and 
generate  knowledge,  e.g.,  “You  should  go  back  and  see  if  there  is  ...”. 

•  Might  act  in  a  role  as  teacher  gently  pushing  collaborative  effort  certain  way 

•  “Contrarians”  when  a  person  says  “Let’s  re-evaluate/  reconsider 

-  Person  disagrees  with  the  current  thinking  of  the  team 

-  “Outlier”  who  makes  the  team  consider  another  viewpoint,  or 

-  “Pulls  back  the  reins” 
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Excerpt  from  MIO  Scenario  Communications  Coding: 
Developing  Solution  Alternatives 


MIO  Team  Communications 

Cognitive  Process  Coding 

Speaker 

Code 

1 

DTR 

A 

Cesium  137  can  be  used  to  make  an  ROD. 
If  there  are  no  explosives,  then  it  is  not 
configured  as  a  weapon  yet.  Recommend 
material  be  confiscated. 

sa 

Develop,  rationalize  and  visualize 
solution  alternatives^  using  data  to 
justify  a  solution 

2 

BO 

Roger  will  confiscate. 

itk 

Individual  task  knowledge  development; 
individual  TM  clarifying  data. 

3 

BO 

Make  sure  you  handle  carefully.  Cs- 
137  is  an  external  gamma  hazard. 

kio 

Knowledge  interoperability:  TMs 
exchanging  knowledge  among  each  other. 

4 

BO 

Roger.  Will  take  precautions. 

kio 

Knowledge  interoperability:  TMs 
exchanging  knowledge  among  each  other. 

5 

soc 

OM 

Does  CG  ship  have  proper  storage 
area  for  material  confiscated? 

itk 

Individual  task  knowledge  development: 
individual  TM  clarifying  data,  asking  for 
clarification. 

6 

soc 

OM 

Search  team  will  report  size  of 
material  and  its  current  containment 
condition;  then  make 
recommendations. 

Cc 

u 

Team  integration  of  individual  TM 
knowledge  for  common  understandings 
one  or  more  TMs  combine  individual 
pieces  of  knowledge  to  achieve  ^  -j 

common  understanding. 

MIO  Scenario  Communications  Coding:  Knowiedge 
interoperability  Development  and  Agreement  on  a  Final  Plan 


\l  fUlMANTlA  n»  H 


MIO  Team  Communications 

Cognitive  Process  Coding 

Speaker 

Code 

BO 

Negative  for  explosives  Station 
2. 

MCkio 

Knowledge  interoperability',  TMs  ex¬ 
changing  knowledge  among  each  other. 

LLNL 

Finally  received  RAD  data 
from  station  2. 

MCkio 

Knowledge  interoperability',  TMs 
exchanging  knowledge  among  each 
other. 

soco 

M 

Will  need  to  resolve  RAD 
containment  hazard  if  it  exists. 

MetCcu 

Team  integration  of  individual  TM 
knowledge  for  common  understandings 
one  or  more  TMs  combine  individual 
pieces  of  knowledge  to  achieve 
common  understanding. 

DTRA 

If  you  have  plutonium,  you 
need  to  confiscate.  It’s  an 
alpha  hazard,  but  still  must  be 
handled  carefully 

MCica 

Iterative  information  collection  and 
analysis;  collecting  and  analyzing 
information  to  come  up  with  a  solution 
but  no  specific  solution  exists. 

BO 

Roger. 

Misc 

Acknowledge  report. 
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MIO  Scenario  Communications  Coding: 

Knowledge  Interoperability  Development  and  Agreement  on 

a  Final  Plan  (cont’d) 


DTRA 

By  the  way,  if  plutonium  is 
in  solid  metal  form,  your 
team  can  handle  safely  with 
rubber  gloves  and  a  dental 
face  mask,  depending  on 
how  much  is  there. 

ica 

Team  shared  understanding 
development  -  discussion  among  all 
team  members  on  a  particular  topic  or 
data  item,. 

BO 

Talking  to  search  team  to 
see  if  this  is  within  their 
capabilities  or  if  we  will 
need  outside  assets. 

kio 

Iterative  information  collection  and 
analysis;  collecting  and  analyzing 
information  to  come  up  with  a  solution 
but  no  svecific  solution  exists. 

LLNL 

Hazard  is  probably 
minimal,  can  isolate  and 
confiscate. 

cs 

Team  agreement  on  a  common  solution 
-  all  tern  members  asree  on  the  final 
plan. 
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Air  Warfare 
Scenarios 

MIO 

Scenarios 

Firefight 

ing 

1 

Macro-Cognitive  Process  Coding 
Categories 

Seen 

D- 

Run 

A 

Seen 

D- 

Run 

B 

CG- 

59 

DDG- 

54 

Nov 

06 

June 

06 

Sept 

06 

Firefight¬ 
ers  9-11 

■ 

Knowledge  Construction 

1. 

Data  to  information  (dti) 

1 

4 

- 

37 

2 

5 

- 

2 

2. 

Individual  mental  model  (imm) 

8 

11 

18 

25 

1 

7 

8 

14 

3. 

Individual  task  knowledge  development  (itk) 

25 

30 

31 

29 

35 

7 

47 

325 

■ 

Team  knowledge  development  (tk) 

11 

5 

18 

1 

3 

5 

8 

210 

5. 

Knowledge  object  development  (ko) 

- 

- 

- 

- 

- 

2 

8 

0 

6. 

Visualization  and  representation  (vrm) 

- 

- 

- 

- 

- 

- 

- 

0 

■ 

Collaborative  Team  Problem  Solving 

7. 

Common  understanding  (cu) 

- 

6 

- 

2 

6 

7 

16 

8. 

Knowledge  interoperability  (kio) 

- 

5 

- 

1 

2 

- 

10 

8 

9. 

Iterative  collection  and  analysis  (ica) 

1 

11 

- 

- 

6 

4 

14 

0 

10 

Team  shared  understanding  (tsu) 

1 

17 

28 

34 

3 

2 

3 

6 

11 

Solution  alternatives  (sa) 

- 

3 

- 

- 

6 

- 

- 

13 

12 

Convergence  of  mental  models  (cmm) 

1 

- 

- 

- 

1 

- 

- 

22 

13 

Agreement  on  Common  solution  (cs) 

- 

2 

- 

- 

- 

- 

1 

Team  Consensus 


Team  negotiation  (tn) 

Team  pattern  recognition  (tpr) 
Critical  thinking  (ct) 

Sharing  hidden  knowledge  (shk) 

Solution  adjustment  against  goal  (sag) 


Outcome  Evaluation  and  Revision 

Compare  solution  options  against  goals 
_ (CSg) _ 

Analyze,  revise  solutions  (aro) 

Miscellaneous  (misc) 

Issue  order  regarding  course  of  action 

(coa) 

Request  take  action  (rta) 


Totals 


MIO  Scenario  Coding  Example 
Team  Shared  Understanding  Development 


MIO  Team  Communications 

DTRA:  Cesium  137  can  be  used  to 
make  an  RDD.  If  there  are  no 
explosives,  then  it  is  not 
configured  as  a  weapon  yet. 
Recommend  material  be 

confiscated. 

BO  Rgr  will  confiscate. 

BO  Mark  material  for  confiscation. 
BO  Make  sure  you  handle 
carefully.  Cs-137  is  an  external 
gamma  hazard. 

BO  Rgr.  Will  take  precautions. 
SOCOM  Does  CG  ship  have 
proper  storage  area  for  mat’l 
confiscated? 

SOCOM  Search  team  will  report 
size  of  material  and  its  current 
containment  condition;  then  make 
recommendations. 


Cognitive  Process  Coding 

sa:  develop,  rationalize  and  visualize 
solution  alternatives  =  using 
data  to  justify  a  solution 
itk:  individual  task  knowledge  dev’t.= 
individual  TM  clarifying  data 
kio:  knowledge  interoperability  =TMs 
exchanging  knowledge  among 
each  other. 

itk:  individual  task  knowledge  dev’t  = 
individual  TM  clarifying  data, 
asking  for  clarification. 
cu:  team  integration  of  individual 

knowledge  for  common  under¬ 
standing  =  one  or  more  TMs 
combine  individual  pieces  of 
knowledge  to  achieve  common 
understanding 
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Model  of  Team  Collaboration: 

Validation 


ksSTv  I  1  4  » M^^SSi 


•  Codes  used  by  Firefighters 

-  19  out  of  the  23  cognitive  processes  in  the  model  (all 
codes  except:) 

•  knowledge  object  development  (ko)-  requires  pictures  and 
icons 

•  individual  visualization  and  representation  of  meaning  (vrm)- 
requires  visuai  aids 

•  iterative  information  coliection  and  anaivsis  (ica)-  coliect  and 
analyze  information  without  mentioning  a  solution 

•  soiution  adjustment  against  aoai  and  exit  criteria  (sag)- 
compares  solution  option  against  goal  and  exit  criteria 

-  Did  not  pertain  to  FDNY  radio  communication  but  still 
pertain  to  other  team  collaboration  environments  and 
should  not  be  eliminated  from  the  collaboration  model 
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Model  of  Team  Collaboration: 

Validation 


Divide  2  hours,  31  minutes  of  recordings  problems  faced 

-  Larger  problem  of  Search  and  Evacuation  -  never  got 
to  final  stages  because  the  buildings  collapsed 

-  Broken  up  into  phases  to  represent  the  mental  model 
within  which  the  FDNY  was  working 

-  Divide  into  smaller  problems 

1.  What  happened?  Create  a  mental  model 
Time  period:  0846-0902 

1.  Evacuate  South  Tower  after  the  North  had  been  hit? 

Time  period:  0902-0958 

1.  How  to  divide  units  between  the  two  towers? 

Time  period:  0958-1028 

1.  Evacuate  the  North  Tower  after  the  South 
collapsed? 

Time  period:  1028-1107 
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Model  of  Team  Collaboration:  Validation 


•  Trends  in  the  codes,  1620  total  speech  turns 

-  849  (52.41%)  miscellaneous,  removed  for  the  following  percentages 
leaving  771  total  codes 

-  325  (42.15%)  itk  -  iterative  team  knowledge  development 

*  asking  lots  of  questions,  how  to  alleviate  questions  and  therefore 
message  traffic? 

-  210  (27.24%)  tk  -  Developing  team  knowledge 

*  Sharing  knowledge  with  fellow  firefighters  and  passing  knowledgi 
back  to  the  dispatcher 

-  92  (11 .93%)  coa  -  Course  of  action 

*  Telling  the  dispatcher  and/or  other  responding  units  what  to  do 

-  53  (6.87%)  rta  -  Request  take  action 

*  Requesting  something  of  the  dispatcher  or  responding  units 

-  22  (2.85%)  cmm  —  Constructing  team  mental  model 

-  16  2.08%)  cu  -  Developing  common  understanding 

-  14  f1 .82%)  imm  -  individual  mental  model 

*  Individuals  contributing  to  the  team’s  mental  model 
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Model  of  Team  Collaboration:  Validation 


*  Inter-rater  Reliability  Analysis 

-  Two  coders,  test  subjectivity  of  model’s  codes 

-  34  out  of  1626  codes  (4.37%)  were  disagreements 

*  Discussed  differing  opinions  to  reach  an 
agreement  with  the  other  coder 

-  49  out  of  1626  codes  (6.31%)  were  decided  upon  after 
a  discussion  between  the  coders 

*  One  or  both  of  the  coders  was  unsure  of  how  to 
code  the  communication  turn  and  left  it  to  discuss 
further  with  the  other  coder. 

*  In  total,  did  not  completely  agree  on  10.68%  of  codes 

*  Reliable  89.32%  of  the  codes 
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Adherence  to  SOP 


*  Minor  Deviations,  SOP  Deviation  #1 

-  iD  speaker  and  addressee 

-  Requesting  ambuiances  and  units 

-  Casuai  communication 

-  10  codes 

*  Major  Deviations,  SOP  Deviation  #2 

-  Unit  to  Unit  transmissions 

-  Use  first  names 

*  Major  Deviations,  SOP  Deviation  #3 

-  Urgent  Radio  Messages  (24  messages  identified 
as  urgent) 

-  Mayday  Radio  Messages  (3  messages  identified 
as  mayday) 

*  Department-wide  Recaii 

-  Never  used  before 

-  Unciear  as  to  where  to  go,  what  to  do 
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Discussion 

Differences  between  three  scenarios  -  how  the  team’s  behavior  maps  to  the  model 

-  Course  of  action  selection  is  done  less  collaboratively  in  tactical  domains, 
due  to  inherent  time  pressure  to  make  decisions  and  take  actions 

•  Decisions  made  unilaterally  by  tactical  action  officer  or  commanding  officer  -- 
do  not  typically  involve  discussion  with  the  rest  of  the  team. 

•  Decisions  regarding  course  of  action  selection  entailed  very  little  collaboration  for 
air  warfare  tasks  due  to  the  speed  of  the  potential  threat  aircraft. 

-  When  actions  need  to  be  taken  very  quickly  in  an  attempt  to  determine  the  intent 
of  an  inbound  track,  time  is  not  available  to  discuss  alternative  courses  of  action 

Air  warfare  consists  of  situation  assessment  (“what’s  going  on”)  and  action  selection 
(“what  to  do  about  it”) 

-  Decisionmakers  use  a  recognition-primed  decisionmaking  strategy  (Klein,  1989) 

•  Situation  itself  either  determines  or  constrains  the  response  options 

•  Recognition  primed  model  of  decisionmaking  fuses  two  processes  — 
situation  assessment  and  mental  simulation  (Klein,  1993) 

-  Simplest  case  the  situation  is  recognized  as  familiar  or  prototypical, 
using  feature  matching,  and  the  obvious  response  is  implemented 

-  More  complex  case  --  decisionmaker  performs  conscious  evaluation  of 
response,  using  mental  simulation  to  uncover  problems  prior  to 
implementing 

-  In  most  complex  case  --  evaluation  reveals  flaws  requiring  modification, 
or  option  is  judged  inadequate/rejected  in  favor  of  next  typical  reaction 

Experienced  DMs  make  90%  of  all  decisions  w/o  considering  alternatives 

•  If  situation  appears  similar  to  one  previously  experienced,  pattern  will  b§^ 
recognized  and  COA  is  usually  immediately  obvious 


Adherence  to  SOP-  Mayday 


Type  of  Mayday  Message 

FDNY  Communications 

Speaker 

Imminent  collapse  feared 

N/A 

Structural  collapse  has  occurred 

FIELD 

Engine  3-9  acting,  report  on  the  22nd  floor, 
reporting  a  floor  collapse  at  that  location,  K. 

A  firefighter  is  unconscious  or 
suffers  a  life  threatening  injury 

FIELD 

We  have  a  medical  emergency,  possible 
heart  attack,  firemen,  we’re  on  the  bulkhead, 
west,  requesting  oxygen  for  the  firemen,  K. 

A  firefighter  becomes  aware 
that  another  firefighter  is  missing 

N/A 

A  firefighter  becomes  trapped  or  lost 

A  civilian  came  on  the  radio  asking  for  help 
because  they  were  trapped  in  the  rubble  after 
the  South  Tower  collapsed.  While  the 
civilian  did  not  know  about  the  correct  use  of 
“mayday”  the  dispatcher  relayed  the  message 
saying,  “transmitting  a  mayday.” 
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Adherence  to  SOP-  Urgent 


Type  of  Urgent 
Message 

FDNY  Communications 

Speaker 

A  firefighter  suffers 
an  injury  that  is  not 
life  threatening,  but 
requires  medical 
attention  and 
hospital  care 

DISPATCH 

Manhattan  to  Field  Comm.,  urgent. 

FIELD 

Receive,  Manhattan,  Field  Comm. 

DISPATCH 

Tower  No.  2,  19*  floor,  firefighter  down.  Tower  No.  2,  19*  floor,  firefighter  down. 

FIELD 

Field  Comm.  Received. 

Discovery  of  a 
structural  problem 
indicating  the  danger 
of  collapse 

FIELD 

Engine  3-9  acting,  report  on  the  22nd  floor,  reporting  a  floor  collapse  at  that  location,  K. 

FIELD 

Marine  1  to  Manhattan  with  an  urgent  message,  K. 

DISPATCH 

Unit  with  an  urgent  message,  K. 

Fire  is  entering  an 
exposure  to  a  degree 
that  any  delay  may 
considerably  enlarge 
the  fire  problem 

FIELD 

This  is  Marine  1,  we’re  in  the  river.  You’ve  got  fire  out  of  the  north  side  and  now 
coming  out  of  the  west  side  of  the  World  Trade  Center,  the  west  side. 

Report  of  apparatus 
breakdown  while 
unit  is  responding  to 
an  alarm 

FIELD 

Engine  317  to  Manhattan,  urgent. 

DISPATCH 

Engine  3-1-7,  go. 

FIELD 

I’ve  got...  from  the  Port  Authority  telling  me  that  the  elevators  are  on  the  44*  floor. 
Don’t  use  them,  they’re  about  to  come  down. 

Loss  of  water  which 
would  endanger 
firefighters 

DISPATCH 

Engine  33  urgent,  go. 

FIELD 

Engine  22  is  being  manned  by  an  off-duty  member  form  Rescue  1 .  Be  advised  it  appears 
that  we  have  lost  water  pressure  down  in  lower  Manhattan.  Can  you  have  Ma^e  1  or 
any  other  available  fire  boat  respond  to  Vescey  Street  on  the  West  Side?  We're  going  to 
need  water  supply  into  the  area,  K. 

Loss  of  SA 


•  Minor  Losses 

-  Vague,  inaccurate  information 

*  Reporting  floor  numbers 

*  Referring  to  the  two  towers 

•  Major  Losses 

-  After  the  South  Tower  collapsed 

*  Who  survived?  Fieid  Comm? 

-  After  the  North  Tower  coiiapsed 

*  Where  were  the  responding  units?  Who  was  in 
which  tower? 

Made  rescuing  those  trapped  very  difficult 
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Loss  of  SA 


SITUATIONAL  AWARENESS  LOST 

FIX 

EFFECTS 

TIME 

SPEAKER 

MESSAGE 

0904 

Marine  6 

Marine  6,  that  plane  was  a  large  bomber-style  green 
aircraft  into  the  second  tower,  be  advised. 

None. 

None. 

0913 

Car  9 

Car  9  to  Manhattan. 

Car  9  came 
back  on  the 
radio  to  correet 
Dispatch,  saying 
they  wanted  the 
mobile  eommand 
vehicle,  not  field 

com 

None,  eorreeted  before  it 
had  an  impact. 

DISPATCH 

Car  9  go  ahead. 

Car  9 

Would  you  advise  the  mobile  command  vehicle  to  come  in 
on  West  and  Liberty  Street,  West  and  Liberty  Street. 

DISPATCH 

I  already  advised  them. 

Car  9 

What’s  their  ETA? 

DIS¬ 

PATCH 

Manhattan  calling  Field  Comm. 

0930 

Engine 

317 

Tve  got ...  from  the  Port  Authority  telling  me  that  the 
elevators  are  on  the  44th  floor.  Don't  use  them, 
they're  about  to  come  down. 

None. 

The  firefighters  work¬ 
ing  in  whichever  build¬ 
ing  the  elevators  were 
not  coming  down  in 
would  have  had  con¬ 
tinued  access  to 
elevators,  but  instead 
were  told  not  to  use 
them  because  of  ^^gue 
information. 

DIS¬ 

PATCH 

Is  that  going  to  be  for  No.  2  or  No.  1 
World  Trade? 

WasnT  sure.  I^d  say  go  with  both. 

DIS¬ 

PATCH 

Attention  all  companies  operating  at  the  fifth  alarm  for  both 
World  Trade  Centers,  the  elevators,  the  Port  Authority 
reports  the  elevators  on  the  No.  4-4  floor  are  about  to  come 
down.  All  companies  operating  at  No.  1  and  No.  2  World 
Trade  Center  at  the  fifth  alarm,  do  not  use  the  elevators. 

They  are  about  to  come  down  as  per  the  Port  Authority  on 
the  No.  4-4  floor.  Field  Comm.,  receive  that  urgent? 
Manhattan  to  Ladder  2-1,  K. 
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